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TABLEAU SEQUENCES, OPEN DIAGRAMS, AND BAXTER FAMILIES 


SOPHIE BURRILL, JULIEN COURTIEL, ERIC FUSY, STEPHEN MELCZER, AND MARNI MISHNA 


Abstract. Walks on Young’s lattice of integer partitions encode many objects of algebraic and combina¬ 
torial interest. Chen et al. established connections between such walks and arc diagrams. We show that 
walks that start at 0, end at a row shape, and only visit partitions of bounded height are in bijection with a 
new type of arc diagram - open diagrams. Remarkably two subclasses of open diagrams are equinumerous 
with well known objects: standard Young tableaux of bounded height, and Baxter permutations. We give 
an explicit combinatorial bijection in the former case. 


1. Introduction 

The lattice of partition diagrams, where domination is given by inclusion of Ferrers diagrams, is known 
as Young’s lattice. Walks on this lattice are of significant importance since they encode many objects of 
combinatorial and algebraic interest. 

At a first level, a walk on Young’s lattice is a sequence of Ferrers diagrams such that at most a box is 
added or deleted at each step. A class of such sequences is also known as a tableau family. There are several 
combinatorial classes in explicit bijection with tableau families ending in an empty shape, in particular when 
there are restrictions on the height of the tableaux which appear. 

In this work we study tableau families that encode walks in Young’s lattice that start at the empty 
partition and end with a partition composed of a single part: A = (to), m > 0. Additionally, they are 
bounded, meaning that they only visit partitions that have at most k parts, for some fixed k. In particular, 
we generalize the results of Chen et al. [T2], and Bousquet-Melou and Xin [7] to prove that two classic 
combinatorial classes -Young tableaux of bounded height and Baxter permutations - are in bijection with 
bounded height tableau families. 

1.1. Part 1. Oscillating tableaux and Young tableaux of bounded height. The first tableau family 
that we study is the set of oscillating tableaux with height bounded by k. These appear in the study of 
partitions avoiding certain nesting and crossing patterns |12j . Our first main result is a bijection connecting 
oscillating tableaux to the class of standard Young tableau of bounded height. Young tableaux are more com¬ 
monly associated with oscillating tableau, and ours is a very different connection. This result demonstrates 
a new facet of the ubiquity of Young tableaux. 

Theorem 1. The set of oscillating tableaux of size n with height bounded by k, which start at the empty 
partition and end in a row shape X = ( m ), is in bijection with the set of standard Young tableaux of size n 
with height bounded by 2k, with m odd columns. 

The proof of Theorem [l] is an explicit bijection between the two classes. One consequence of the bijective 
map is the symmetric joint distribution of two kinds of nesting patterns inside the class of involutions. 

Enumerative formulas for Young tableaux of bounded height have been known for almost half a century |181 
mm, but new enumerative formulas can be derived from Theorem [lj notably an expression which can be 
written as a diagonal of a multivariate rational function. The new generating function expressions are the 
subject of Section |4.3[ 

1.2. Part 2. Hesitating tableaux and Baxter Permutations. In the second part, we consider the 
family of hesitating tableaux. These tableau sequences appear in studies of set partitions avoiding so-called 
enhanced nesting and crossing patterns. The work of |12j again serves to describe bijections between lattice 
paths and arc diagrams. Using a lattice path interpretation, we make a generating function argument 

Key words and phrases. Young tableaux, nonnesting partitions, matchings, Baxter permutations, bijections, oscillating 
tableaux. 
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to connect this combinatorial class to Baxter permutations. This connection was recognized by Xin and 
Zhang [3Sj. Here we offer an explicit proof, using formulas of Bousquet-Melou and Xin [7], of the following 
result. 

Theorem 2. The number of hesitating tableaux of length 2 n of height strictly less than three is equal to the 
number B n+ i of Baxter permutations of length n + 1, where 

n /n+l\ /n+l\ /n+l\ 

o _ U-lA k Afc+1/ 

n ~h mm ' 

Baxter numbers have been described as the “big brother” of the well known Catalan numbers: they are the 
counting series for many combinatorial classes, and these classes often contain natural subclasses which are 
counted by Catalan numbers. For example, doubly alternating Baxter permutations have a Catalan number 
counting sequence m- One consequence of Theorem [2] is a new two variable generating tree construction 
for Baxter numbers. 

Unlike the results in Part 1, our proof of Theorem [2] is not a combinatorial bijection. One impediment to 
a bijective proof is a lack of a certain symmetry in the class of hesitating tableaux that is present in most 
known Baxter classes. A bijection would certainly be of interest, and in fact we conjecture a refinement of 
Theorem [2j in Conjecture [TJ which could guide a combinatorial bijection. 

We begin with definitions in Section [21 and some known bijections. Then we focus on the standard Young 
Tableaux of bounded height in Section [4| followed by our study of Baxter objects in Section [bj 

2. The combinatorial classes 

We begin with precise definitions for the combinatorial classes that are used in our results. 

2.1. Tableaux families. As mentioned above, a common encoding of walks on Young’s lattice is given by 
sequences of Ferrers diagrams. We consider three variants. Each sequence starts from the empty shape, and 
has a specified ending shape; the difference between them is the limitations they impose on when one can 
add or remove a box. The length of a sequence is the number of elements, minus one. (It is the number of 
steps in the corresponding walk.) 

A vacillating tableau is an even length sequence of Ferrers diagrams (A*- 0 ),..., where consecutive 

elements in the sequence are either the same or differ by one square, under the restriction that 
A (2*) > X (2i+i) and A (2i-i) < X (2i) 

A hesitating tableau is an even length sequence of Ferrers diagrams (A^°\..., A^ 2 ™)) where consecutive 
differences of elements in the sequence fall under one of the following categories^ : 

• A^ 2 *) = A^ 2l+1 ) and A^ 2l+1 ) < A^ 2l+2 ) (do nothing; add a box) 

• A (2i ) > A( 2i+1 )and A^ 2i+1 ) = A< 2i+2 ) (remove a box; do nothing) 

• A (2i) < A (2i+1) and A (2i+1) > A (2i+2) (add a box; remove a box). 

An oscillating tableau is simply a sequence of Ferrers diagrams such that at every stage a box is either 
added or deleted. Remark that the length of the sequence is not necessarily even. 

In each case, if no diagram in the sequence is of height k + 1, we say that the tableau has its height bounded 
by k. Figure [l] shows examples of the different tableaux. 

2.2. Lattice walks. Each integer partition represented as a Ferrers diagram in a tableau sequence can also 
be represented by a vector of its parts. If the tableau sequence is bounded by k. then a /c-tuple is sufficient. 

The sequence of vectors defines a lattice path. For example, each of the three tableau families above each 
directly corresponds to a lattice path family in the region 

W k = {(x 1 ,x 2 , ■ ■ ■ ,x k ) : Xi € Z,x 1 > x 2 > ■ ■ ■ > x k > 0} 

starting at the origin (0,..., 0). We can explicitly define three classes of lattice paths by translating the 
constraints on the tableau families. 


1 Recall A < means that A i < ^ for all i 
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(0,0, □ , □ , 0 , □ , mu , cm , me, me, 
(0,0, □, □, m, p, g, □, □) 
(0, □, g, fp m, p, gg, gp, gp, mm 


m 

■ mm) 


Figure 1. From, top to bottom. A vacillating tableau of length 10, a hesitating tableau of 
length 8, an oscillating tableau of length 11. In each case, the height is bounded by 2. 




Figure 2. From left to right. A IFVvacillating walk, a W^-hesitating walk, a WVoscillating 
walk. The stay steps are drawn as loops. (These walks correspond to the tableaux of 
Figure [l]) 


Remark. Twice in this article, in order to relate previous results, we use a translation of this region and 
still identify it as Wk■ The translated regions are identical to the original up to a small shift of coordinates. 
This change is detailed explicitly in the text (the allowed sets of steps are never changed). 

Let ei be the elementary basis vector with a 1 at position i and 0 elsewhere. The steps in our lattice 
model are all elementary vectors, with possibly one exception: the zero vector, also called stay step. The 
length of the walk increases with a stay step, but the position does not change. 

A Wfc-vacillating walk is a walk of even length in Wk using (i) two consecutive stay steps; (ii) a stay step 
followed by an step; (iii) a — e* step followed by a stay step; (iv) a —step followed by an step. 
A Wfc-hesitating walk has even length and steps occur in the following pairs: (i) a stay step followed by 
an e* step; (ii) a — e; step followed by a stay step; (iii) an e* step follow by —ej step. 

A Wfc-oscillating walk starts at the origin and takes steps of type et or —e*, for 1 < i < k. It does not 
permit stay steps. 

Some examples are depicted in Figure [2j 

2.3. Open arc diagrams. Arc diagrams are a useful way to provide a graphical representation of a com¬ 
binatorial class. They are particularly useful to detect certain patterns. Matchings and set partitions are 
examples of classes that have natural representations using arc diagrams. In the arc diagram representation 
of a set partition of {1,2,..., n}, a row of dots is labelled from 1 to n. A partition block {ai, 02 ,..., aj}, 
ordered a\ < 02 < ... < aj , is represented by the set of arcs {(cii, CJ 2 ), (« 2 j 0,3), ..., (dj-i, aj)} which are 
always drawn above the row of dots. We adopt the convention that a part of size one, say {*}, contributes 
a loop, that is a trivial arc In this work, we do not draw the loops, although some authors do. The 

set partition n = {{1,3, 7}, {2,8}, {4}, {5,6}} is depicted as an arc diagram in Figure [3j Matchings are 
represented similarly, with each pair contributing an arc. 

A set of k distinct arcs (*i, Ji),..., (ik,jk) forms a k-crossing if i\ < 12 < • • • < ik < ji < J 2 < 11 • < jk- 
They form an enhanced k-crossing if i\ < i 2 < • ■ • < ik < ji < J 2 < • • • < jk- (By convention, an isolated 
dot of the partition forms an enhanced 1-crossing.) They form a k-nesting if i\ < 12 < ■ • • < ik < jk < ■ • ■ < 
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Figure 3. The set partition n = {1,3,7}, {2,8}, {4}, {5,6} 

J 2 < j i- They form an enhanced k-nesting if i\ < i^ < ■ ■ ■ < ik < jk < ■ • • < j 2 < j l (As previously, ik = jk 
means that ik is an isolated element in the set partition.). Figure [4] illustrates a 3-nesting, an enhanced 
3-nesting, and a 3-crossing. 




Figure 4. Patterns in arc diagrams. From left to right: a 3-crossing, an enhanced 3- 
crossing, a 3-nesting, an enhanced 3-nesting. 

Recently, Burrill, Elizalde, Mishna and Yen m generalized arc diagrams by permitting open arcs : in 
these diagrams each arc has a left endpoint but not necessarily a right endpoint. The open arcs can be 
viewed as arcs “under construction”. An open partition (resp. an open matching ) is a set partition (resp. a 
matching) diagram with open arcs. In open matchings, the left endpoint of an open arc is never the right 
endpoint of another arc. Figure [5] shows examples of such diagrams. 



123456789 123456789 10 


Figure 5. An open partition and an open matching. 


We are also interested in crossing and nesting patterns in open diagrams. Here we simplify the notation 
°f [TO] - A k-crossing in an open diagram is either a set of k mutually crossing arcs (as before), or the union 
of k — 1 mutually crossing arcs and an open arc whose left endpoint is to the right of the last left endpoint 
and to the left of the first right endpoint of the k — 1 crossing arcs. A k-nesting in an open diagram is either 
a set of k mutually nesting arcs, or a set of k — 1 mutually nesting arcs, and an open arc whose left endpoint 
is to the left of the k — 1 nesting arcs. We generalize enhanced fc-crossings and enhanced /c-nestings in an 
open diagram similarly. Examples are given in Figure [6] If we want to point out that a crossing (or nesting) 
has no open arc, we say that it is a plain fc-crossing (or fc-nesting). 



Figure 6. Patterns in open diagrams. From left- to right: a 3-crossing, an enhanced 3- 
crossing, a 3-nesting, and an enhanced 3-nesting. 


3. Bijections 

3.1. Description of Chen, Deng, Du, Stanley, Yan’s bijection. The work of Chen, Deng, Du, Stanley 
and Yan m describes bijections between arc diagram families and tableau families. In this section we 
summarize a selection of their results, and adapt it to our needs. Their main bijection maps a set partition 7 r 
to a sequence of Young tableau^ the shapes of which form a vacillating tableau, denoted by </)(7r). We do 

2 A Young tableau is defined here as the filling of a Ferrers diagram with positive integers, such that the entries in each row 
and in each column are strictly decreasing (usually the entries are increasing; the reason for this change is explained later). The 
set of entries does not need to form an interval of the form {1,..., n}. 
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Figure 7. Top. The set partition tt = {1, 5}, {2,4, 6}, {3}. Middle. The corresponding 
Young tableau sequence. Bottom. The vacillating tableau given by <j>( tt). 


not describe the generalization of their construction to hesitating tableaux and oscillating tableaux (still due 
to Chen et ai), but it exists and it will be used for the proof of Propositions [6] and [7j 

We describe here their bijection (f> but with a slight difference: we read the arc diagrams from left to 
right, instead from right to left as it was done originally. In concrete terms, it means that the image of a 
partition tt under </>, as we write it, is the mirror image of the actual 4>(tt). Our approach is justified by the 
fact that natural properties emerge when the reading direction is swapped. This can be particularly seen 
through Proposition [3j where the size of the crossings and nestings around the ith dot is linked to the height 
and the width of the itli Ferrers diagram. 

Let tt be a set partition of size n. We are going to build from tt a sequence of Young tableaux where 
the entries are decreasing in each row and each column - the fact that we use decreasing order instead of 
increasing order is a direct consequence of the change of the reading direction. The first entry is the empty 
Young tableau. We increment a counter i by one from 1 to n. A given step in the algorithm proceeds as 
follows. If i is the right-hand endpoint of an arc in n, then delete i from the previous tableau (it turns 
out that i must be in a corner). Otherwise, replicate the previous tableau. Then, after this move, if i is a 
left-hand endpoint of an arc (i,j) in tt, insert j by the Robinson-Schensted-Knuth (RSK) insertion algorithm 
for the decreasing order into the previous tableau. If i is not a left-hand endpoint, replicate the previous 
tableau. 

The output of this process is a sequence of Young Tableaux starting from and ending at the empty Young 
tableau. The sequence of shapes is given by a vacillating tableau and is denoted 4>(tt). 

Example. Consider the partition tt from Figure[7j The number 1 is the left-hand endpoint of the arc (1,5), 
but not the right-hand endpoint of any arc, so the first three Young tableaux are 0, 0, 0. Similarly, 2 is the 
left-hand endpoint of (2,4) but not a right-hand endpoint, so the two following Young Tableaux are 0 , 00 . 
The number 3 is an isolated point, so the tableau 15141 is repeated twice. The number 4 being the right-hand 
endpoint of (2,4) and the left-hand endpoint of (4, 6), we delete 4, then we add 6: we obtain 0, |lj The rest 

of the sequence is given in Figure [TJ 

Given a vacillating tableau (0, Ai,..., A 2 ra -i, 0), there exists a unique way to fill the entries of the Ferrers 
diagrams into Young tableaux so that it corresponds to an image of a set partition. This has been proved 
in [l2j . and implies that <f> is a bijection. 

In an arc diagram, we say that the segment [*, i- 1-1] is below a fc-crossing if the arc diagram contains k arcs 
(ii, ji),..., ( ik,jk ) such that i± < «2 < • • • < ik < i and i + 1 < ji < j 2 < • • • < jk • Similarly, the segment 
[i,i + 1] is below a fc-nesting if there exist k arcs («i,ji),..., ( ik,jk ) such that i\ < 12 < • • • < ik < * and 
i + 1 < jk < • < j -2 < ji- For instance, in Figure [7j the segment [3,4] is below a 2-nesting but not below a 

2-crossing, while the segment [4, 5] is below a 2-crossing but not below a 2-nesting. With this definition we 
can formulate and prove a stronger version of B3 Theorem 3.2] (this property can also easily be seen in the 
growth diagram formulation of the bijection - see [22]). 


5 



























Proposition 3. Let n be a partition of size n and <P(tt) = (Ao,..., A 2n ). For every i £ {1 the 

segment [i,i +1] of tt is below a k-crossing (resp. k-nesting) if and only if \ 2 i in </>(7r) has at least k rows 
(resp. k columns). 

Example. We continue our example and verify that A^ = I I has 2 columns but not 2 rows, and 
accordingly [3,4] is below a 2-nesting, but not a 2-crossing. 

Proof. Let (To,..., T 2n ) be the sequence of Young tableaux corresponding to the partition 7r. We use some 
ingredients from the proof of Theorem 3.2 of [H p. 1562Q 

(1) A pair (i,j) is an arc in the representation of tt if and only if j is an entry in T 2 i,T 2 i + i,... ,T 2 (j_iy, 

(2) Let at = w±W 2 ■ ■ ■ w r denote the permutation of the entries of T( such that w\, W 2 , ■ ■ ■, w r have been 
inserted in (T 0 ,..., T 2n ) in this order; 

(3) The permutation eq has an increasing subsequence of length k if and only if the partition A, has at 
least k rows. 

The following statements are then equivalent: 

• The segment [i, i + 1] is below a fc-crossing. 

• There exist k arcs (*i, ji), ■ ■ ■, (ikijk) i n tt such that 

i\ < h < ■ ■ ■ < ik < * and i + 1 < j\ < j 2 < ■ • • < j k . 

■ There exist k numbers j\ < j 2 < ■ ■ • < jk that are entries of T 2i such that j-\. j 2 ,... ,jk have been 
inserted in this order in (To,..., T 2ra ). 

• There exist k numbers ji < J 2 < • • • < jk such that J 1 J 2 • • ■ jk is a subsequence of <t 2 ,;. 

• The diagram A 2 i has at least k rows. 

The proof for fc-crossings is similar. □ 


Considering all intervals [i,i + 1] for 1 < i < n, we recover the statement of Theorem 3.2 from [12]- 

Corollary 4. A set partition tt has no (k + l)-crossing (resp. no (k + 1 )-nesting) if and only if no Ferrers 
diagram in the sequence 4>(tt) has k + 1 ivws (resp. columns). 


Remark. The crossing level of a set partition 7r, denoted cr(7r), is the maximal k such that tt has a fc- 
crossing. Similarly, the nesting level of a set partition tt, denoted ne{ tt), is the maximal fc such that tt has a 
fc-nesting. Chen et al. conclude from the previous corollary that the joint distribution of cr and ne over all 
the set partition diagrams of fixed size is symmetric. That is, 


E 


x cr(Tr) yne(n) 


7r set partition diagram 
of size n 


E 


y Cr M X ne ^) 


7r set partition diagram 
of size n 


Let r denote transposition, the operation that transposes every Ferrers diagram inside a vacillating tableau. 
Then </)~ 1 oTO(j) swaps the crossing level and the nesting level of a set partition. Moreover, note that ^ -1 oto^> 
preserves the opener/closer sequence, i.e., if the number i is an isolated point (resp. a left endpoint, a right 
endpoint, a left and right endpoint at the same time) in a partition tt, then i is an isolated point (resp. a 
left endpoint, a right endpoint, a left and right endpoint at the same time) in <^> -1 o r o <f>(q r). 


3.2. Bijections with open partitions. Next we describe a generalization of the bijection of Chen et al. 
to the class of tableaux ending at a row shape. We thereby link to the classes of Section [2] 

Proposition 5. A bijection can be constructed between any two of the following classes: 

(1) the set of open partition diagrams of length n with no (fc + 1)-crossing, with m open arcs; 

(2) the set of open partition diagrams of length n with no (fc + 1 )-nesting, with m open arcs; 

(3) the set of vacillating tableaux of length 2 n, with maximum height bounded by k, ending in a row of 
length m; 

(4) the set of Wk-vacillating walks of length 2 n ending at (to, 0,..., 0). 
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Recall that one bijection is the mirror image of the other. So the indices differ between E3 and here. 
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Proof. Bijection (1) (3). Since we would like to use the aforementioned bijection </>, the idea here 

simply consists in closing every open arc in a very natural way - we thereby recover classic closed diagrams. 

Let us be more precise. Let 7 r be an open partition diagram of length n with to open arcs and no ( k + 1)- 
crossing. We build a new partition diagram 7f of length n+m without open arcs by closing the to open arcs of 
7T in decreasing order. That is, if *i < i 2 < • ■ ■ < i m denote the positions of the to open arcs of 7r, the partition 
7f is the closure, obtained by replacing the to open arcs with the arcs (ii, n+m), (* 2 , n+m — 1),..., ( i m , n+1), 
as shown in Figure [8| Note that the open arcs are closed in such a way that no new crossing is created. 



• • • • • •• •• 7]- = • 9 9 9 9 •* •••»• 

123456789 123456789 10 11 12 


Figure 8. Left. An open partition diagram, 7r, with 3 open arcs. Right. The corresponding 
closed partition diagram 7f ending with a 3-nesting, obtained by closing the 3 open arcs of 
7T in reverse order. 

The to last elements of 7f form the end of an m-nesting. Consequently, each crossing of n has at most one 
element inside {n + 1,..., n + to}; so the preimage of any ^-crossing of 7f is also an ^-crossing. As 7 r has no 
(k + l)-crossing, the diagram W has no (k + l)-crossing. 

Let 4>(Tf) = (Ao,..., A 2 ( n + m )) be the image of 7f under (j). By Corollary |4j the height of this vacillating 
tableau is bounded by k. Moreover, the segment [n, n + 1] in 7f is below an m-nesting but not below a 
2-crossing. By Proposition [ 3 ] it means that A 2n is a column with at least m rows. Since 4>(tt) ends with 
an empty diagram and one can delete at most one cell every two steps, A 2rt has exactly to rows. Thus, 
(Ao,..., A 2 n) is a vacillating tableau of length 2n, with maximum height bounded by k, ending in a column 
of length m. 

The transformation is bijective: a vacillating tableau (Ao,..., A 2n ) from the set (3) can be concatenated 
with ((to — 2), (to — 2),...,(1),0,0), where (j) denotes the partition of i only composed of a single part of 
size j. If we change its preimage under (f> by opening the arcs ending in {n + 1,..., n + to} into to open arcs, 
we recover the initial open diagram 7r. 

Bijection (2) -4+- (3). The previous bijection is adapted with an additional application of the transpo¬ 
sition operator r. 

Bijection (3) -4+- (4). This is a straightforward consequence of the encoding. As the vacillating tableaux 
end at a row of length to, the endpoints of the walks must be the point (to, 0,..., 0) □ 

The open diagram case inherits many properties from the closed diagram case. For example, the statistics 
of crossing level and nesting level are equidistributed. Also, the problem of finding a direct bijection between 
open partitions with no fc-crossing and open partitions with no fc-nesting without going through the vacillating 
tableaux seems to be as difficult as the closed case. 

However, the nesting level and the crossing level do not have symmetric joint distribution for open 
partitions. This constitutes a difference with the (closed) partition diagrams. 

Furthermore, the other generalizations of Chen et al. - specifically the ones that concern the hesitating 
and oscillating tableaux - can also be extended to tableaux ending at a row shape, and open partitions. The 
proofs are similar. 

Proposition 6. The following classes are in bijection: 

(1) the set of open matching diagrams of length n with no (k + 1)-crossing, with to open arcs; 

(2) the set of open matching diagrams of length n with no (k + 1 )-nesting, with to. open arcs; 

(3) the set of oscillating tableaux of length n, with height bounded by k, ending in a row of length to; 

(4) the set of Wk-oscillating walks of length n ending at (to, 0,..., 0). 

Proposition 7. The following classes are in bijection: 

(1) the set of open partition diagrams of length n with no enhanced (k + 1 )-crossing, with m open arcs; 

(2) the set of open partition diagrams of length n with no enhanced (k + l)-nesting, with m open arcs; 
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(3) the set of hesitating tableaux of length 2n, with height bounded by k, ending in a row of length to; 

(4) the set of Wk-hesitating walks of length 2 n ending at (to, 0,..., 0). 


4. Young tableaux, involutions and open matchings 


4.1. Bijections. We can now prove our first main result, namely Theorem[T| Our strategy is to use Propo¬ 
sition [6j and prove the following result, from which Theorem [I] is a straightforward consequence. 


Proposition 8. The set of standard Young tableaux of size n with height bounded by 2k and m odd columns 
are in bijection with the set of open matching diagrams of length n, with m open arcs and with no (k + 1)- 
crossing. 


As far as we can tell, this theorem was first conjectured by Burrill BO Our proof uses the Robinson- 
Schensted-Knuth (RSK) correspondence, and the bijection of Chen et al.. 

A different proof was communicated to us by Christian Krattenthaler [Mj. It relies on the RSK correspon¬ 
dence like our proof, but also on jeu de taquin (an operation on Young tableaux invented by Schiitzenberger (2Sj 
We note that the two bijections differ: our bijection has the advantage of preserving ■ just like the Chen 
et al. construction - the “opener/closer” sequence (in a formulation using diagrams on both sides of the 
bijection; cf Lemma 11 for more details), a strong property which does not clearly appear in Krattenthaler’s 
alternative. His proof passes through growth diagrams [23] , 

The following lemma presents a classic property of the RSK correspondence. 


)• 


Lemma 9. (Robinson-Schensted-Knuth correspondence) The set of standard Young tableaux of size n with 
height bounded by k and m odd columns is in bijection with involutions of size n with m fixed points and no 
decreasing subsequence of length k + 1. 


1 

CO 
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Figure 9. Left. A standard Young tableau Y of size 10. Right. The arc diagram represen¬ 
tation of the involution (1 7) (3 9) (4 6) (5 10). This involution is the image of ( Y,Y ) under 
the RSK correspondence. 


As a first step, Lemma[9]yields combinatorial objects that are close to open matchings. Indeed, involutions 
have a very natural arc diagram representation: cycles (ij) are represented by an arc, and fixed points are 
isolated dots. An example is shown in Figure [9] We can map involutions into the set of open matchings by 
simply changing every isolated point into an open arc. Under this map, there is a simple correspondence 
between decreasing sequences in an involution and nestings in the open diagram. 

Lemma 10. Let k € Z>i. An involution has no decreasing subsequence of length 2k + 1 if and only if there 
is no enhanced k-nesting in its arc diagram representation. 

Proof. Let a be an involution. If its arc diagram has an enhanced fc-nesting then a contains k cycles 
(U ii)> • • •) 0k jk) that satisfy i\ < i% < ■■■ < ik < jk < ■•■ < j i, which clearly induces a decreasing 
subsequence of length 2k — 1. 

Conversely, assume that there exist 2k— 1 numbers ii < *2 < • • ■ < i 2 k-i such that ct(« 2 fc-i) < • • ■ < a(i\). 
If a(ik) — ik > 0, then i\ < ■ ■ • < ik < ot{ik) < ■ ■ ■ < ct(ii): this means that (ii, a(?'i))> • •., {ik, ot{ih)) form 
an enhanced fc-nesting. Otherwise, a{ik) — ik < 0. Thus afefc- 1 ) < • • • < a(ik) < ik < • • • < * 2 fc-u the arcs 
(a(* 2 fe_i), * 2 fc—l), ■ ■ •, (ct(ifc), ik) form an enhanced fc-nesting. □ 

By the two preceding lemmas, the proof of Proposition 8 is reduced to the proof that involution diagrams 
of length n with to fixed points and no enhanced (fc + l)-nesting are in bijection with open matching diagrams 
of length n with to open arcs and no (k + l)-crossing. This is established by the following lemma. 

Aloi'e precisely, this conjecture used open matchings with no (k + l)-nesting. 

















Figure 10. Image of the involution of Figure [ 9 ] under ip. 


Lemma 11. There is a bijection ip between involution diagrams and open matching diagrams, such that for 
a an involution diagram and (3 = ip (a), the diagrams a and (3 have same length, the number of fixed points 
in a is the number of open arcs in (3, and for any £ > 1 there is an enhanced £-nesting in a if and only if 
there is an £-crossing in f3. In addition the opener/closer sequence of a (seeing fixed points as openers) is 
the same as the opener/closer sequence of (3. 


Proof of Proposition [#| We describe ip, a bijective map between involutions and open matchings. It is formed 
as a composition of other maps. We have already defined <p , the bijection from set partition diagrams to 
vacillating tableaux from Section |3.1[ and r, the transpose action which can be applied to any tableau 
sequence. We add l, the operation that changes every isolated dot in an involution diagram into an open 
arc. Let ip be the composition l o </> -1 o r o (j>. Figure 10 shows an example of the action of ip. 

Since <p, r and 1 can all be reversed, the mapping ip is bijective. Moreover, recall from the end of 
the mapping </> -1 o r o cp preserves the opener-closer sequence. Therefore, every involution of 


Section 3.1 


size n with m fixed points is mapped under ip to an open matching diagram of size n with m open arcs. 

Assume that an involution a has an enhanced Anesting (ii, ji)... (if., ft)- If if ^ jt , this enhanced nesting 
is also a plain Anesting. By the remark at the end of Section 3.1 we know that (p^ 1 oTocp(a) has a ^-crossing, 
so the same holds for ip(a). 

If if = ji, then in is a fixed point of a and hence an open arc in ip(a). Moreover, the segment [if,if + 1] 
is below the (£ — l)-nesting (i\,j\) ■ ■ ■ (it-\,jf-\). So by Proposition [3j the 2ifih diagram of cp(a) has at 
least t columns. The 2ifth. diagram of to (p(a) has then k rows, and so (if, if + 1] is below a (£ — l)-crossing 
in ip(a). Thus, if is in ip(a) an open arc below a (l — l)-crossing: the open matching ip(a) has a ^-crossing. 
The converse is proved similarly. 

In summary, ip is a bijection between involution diagrams of size n with to fixed points and no enhanced k- 
nesting and open matchings diagrams of size n with to open arcs and no fc-crossing. □ 


The standard Young tableau in Figure [9] is mapped to the open arc diagram at the bottom of Figure [T0| 
Here, the parameter to takes value 2. 

Remark that standard Young tableaux with height bounded by odd numbers are also characterized in 
terms of open matching diagrams (but this time constrained by the plain nestings or crossings). 


Proposition 12. The following classes are in bijection: 


9 






















(i) the set of standard Young tableaux of size n with m odd columns and height bounded by 2k — 1; 

(ii) the set of involutions of size n with m fixed points and no decreasing subsequence of length 2k; 

(in) the set of open matching diagrams of length n with no plain k-crossing and with m open arcs; 

(iv) the set of open matching diagrams of length n with no plain k-nesting and with m open arcs. 


Proof. The RSK correspondence (specifically, the property described in Lemma [9]) gives a straightforward 
bijection between (i) and (ii). Then, seeing isolated points as open arcs, it is easy to adapt Lemma 10 
order to show the correspondence between (ii) and (iv). Finally the bijection between (iii 
by (j>~ 1 orocj) (still by interpreting isolated points as open arcs), where <j> and r are defined in Section 3.1 


and (iv) is given 
□ 


4.2. A new symmetric joint distribution for involutions. While looking for the previous bijection 
we found a surprising symmetry property for involutions, which is now presented. Section [XT] contained the 
definition of nesting level ; in the context of involution diagrams, the notion can be refined in two different 
ways, depending on whether we regard involution diagrams as enhanced set partition diagrams or as open 
matchings. 

The enhanced nesting level of an involution a, denoted ne^(a), is the maximal number of dots in an 
enhanced nesting of a (note that k marks the number of dots not number of arcs). Pursuant to Lemma 10 
the number ne^(a) is also the length of the longest decreasing subsequence of a. Similarly, we define the 
open nesting level of an involution a , denoted ne<_(a): after transforming the diagram of a into an open 
matching by changing every isolated point into an open arc, the open nesting level of a is the maximal 
number of dots inside a nesting. 

Remark that an enhanced nesting and a nesting in an open diagram are identical if they both have an 
even number of dots; these are then plain nestings. The difference is made when the number of dots is odd, 
say 2k + 1. In this case, an enhanced nesting is made of a dot below a plain fc-nesting, while a nesting in an 
open matching is made of an open arc to the left of a plain fc-nesting. This justifies the notation ne^ and 
ne<_. Figure [TT| compares the two patterns. 

Example 13. The open nesting level of the involution (1 7)(3 9)(4 6)(5 10), depicted in is 5: the 

numbers 2, 3,4,6, 9 form a nesting if we transform the dot 2 into an open arc. However the enhanced nesting 
level of the same involution is 4: there is no dot below any 2-nesting. 



Figure 11. Left. An enhanced 3-nesting with respectively 5 and 6 dots. Right. A 3-nesting 
with respectively 5 and 6 dots (in an open matching). 


A (weak) link between the two statistics can be easily derived from the preceding study, as stated in the 
following proposition. 


Proposition 14. There is a bijection 6 from involution diagrams to involution diagrams such that for a 
any involution diagram, (d = 9 (a), and l > 1, there is an enhanced £-nesting in a if and only if there is an 
t-nesting in the open matching obtained by changing every fixed point in fd by an open arc. In other words, 
there exists a bijection 9 between involutions a such that nej.(a) = 2k — 1 or 2k and involutions /3 such that 
ne«_(/3) = 2k — 1 or 2k. 

In addition, 9 preserves the length, the number of fixed points and the opener/closer sequence (viewing 
fixed points as openers). 


Proof. We define 9 as the composition of 


(1) the mapping <j> described by Lemma 11 

(2) the operation that changes isolated points into open arcs; 

(3) the mapping from open matchings to oscillating tableaux with bounded width ending a column (see 
Subsection 3.2 and more precisely Proposition [6]); 
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(4) the transposition of the Ferrers diagrams; 

(5) the mapping from oscillating tableaux with bounded height ending at a column to open matchings; 

(6) the operation that changes open arcs into isolated points. 

All the properties of 9 presented in the statement of this proposition are direct consequences of Lemma m 
and Proposition [6] □ 

Note that an enhanced nesting is preserved when the diagram is reflected. This is not true for an odd 
nesting in an open diagram, because the isolated point must be to the left of the nesting. Despite the 
fact they do not share this property, the enhanced nesting level and the open nesting level have symmetric 
distribution, as stated in the following theorem. 


Theorem 15. The statistics ne<_ and neq have a symmetric joint distribution over all the involutions of 
size n with m fixed points, i.e., 


E 


j. ne t- (oO y ne l ( a ) 


E 


yne^(a) x n ei (a) 


ol involution 
of size n 

with m fixed points 


ot involution 
of size n 

with m fixed points 


Remark. The bijection 9 from Proposition 14 does not swap the statistics ne^ and ne<_. For instance, 
the involution aq = (1 5)(2 3) of size 5 is mapped to the involution a 2 = (2 3)(4 5): we have nej.(aq) = 
ne«_(aq) = 4 but ne < _(a 2 ) = 3 and (even worse!) ne.j.(a: 2 ) = 2. Nonetheless, the existence of the function 9 
(and more particularly the fact that an involution with enhanced nesting level 2k — 1 or 2k is mapped under 


9 to an involution with open nesting level 2k — 1 or 2k) is sufficient to prove Theorem 15 


Proof. Consider all involutions of fixed size, with a fixed number of fixed points. Let aij be the number of 
involutions a in this class of such that ne^(a) = i and ne<_(a) = j. By Proposition 14 the bijection 9 maps 
involutions a such that neq(a) = 2k — 1 or 2k to involutions /3 such that ne«_(/3) = 2k — 1 or 2 k; hence 


( 1 ) «2fc-lj + a 2k,j — £ + di,2k- 

j >0 i>0 

We can simplify the expression in Equation ([l]) as the values neq(a) and ne<_(a) can only differ by at most 
one for a given involution a. Indeed, if t denotes the maximal number of arcs inside a nesting of an involution, 
the open nesting level and the enhanced nesting level must equal either 2 i or 21 + 1. Therefore, a t _j = 0 
except for pairs (i,j) of the form (2£, 21), (fit, 21 + 1), (2£ + \,2l) or (2£ + 1,2£+ 1). Equation Q can be 
thus rewritten as: 


02k-l,2k-l + 0,2k-l,2k-2 + &2k,2k + 0,2k,2k+l — 0,2k-2,2k-l + 02fc-l,2fc-l + 0,2k,2k + 0,2k+l,2k, 

or after simplification 

02fc,2fc+l — 02fc+l,2fe = 0,2k—2,2k — 1 — 0,2k-l,2k-2- 

In other words, the sequence (a 2 fc, 2 fc+i — a 2 fc+i, 2 fe) is constant over all k > 0. But since it equals 0 for k = 0, 
we have for every k > 0, 

0,2k,2k+l — 0,2k+l,2k- 

The other terms a» j such that i ^ j vanish, so the last equality is sufficient to conclude the proof. □ 

The previous proof is simple but not constructive: can we describe an involution (on involutions) that 
swaps the statistics ne<_ and ne^? The answer is yes, and a description can be given in terms of iterations 
of 9, where 9 is the mapping defined by Proposition [l4| 

Lemma 16. Let 9^ be the £th iteration of 9 and Aij be the set of involutions a such that neq(a) = i and 
ne<_(a) = j. 

For every a in A2k,2k +1 with k > 0, there exists m > 1 such that 9 ^\a) A2k,2k+1 Uiamt for 

£ £ {1 ,... ,m — 1}, and 9 ^(a) £ A2k+i,2k- Moreover, for every a' in A2k+i,2k, there exists m' > 1 such 
that 9 ^'\a') ^ A2k,2k +i U A 2fc+ i j2 fc for £' £ {1 ,..., to 7 — 1 }, and 9 ^ m \&') £ A2k,2k+1- 

In other words, in the orbit of any involution under 9 (this orbit is cyclic since 9 is bijective and the 
set of involutions of fixed size is finite), the elements of A 2 k, 2 k +1 U4 2 q^ 2 (; alternate between A 2 k, 2 k+i and 

A2k+l,2k- 


li 





O QQQ 



Figure 12. Schematic representation of typical orbits under 9. The white circles represent 
the elements of A 2 k, 2 k+i, the gray circles the elements of A- 2 k+i, 2 k and the small points are 
the remaining elements. 


An example of this correspondence is illustrated in Figure [12} 

Proof. Consider i > 0 such that 9^ (a) = a (such an i exists as 6 acts bijectively on the finite set of 
involutions of a fixed length). Since ne < _(a) = 2k + 1, we have ne^(0^ _1 - ) (a)) > 2k. 

Let m denote the smallest j > 0 such nej.(6hU(a)) > 2k. We have then nej_(6^ m-1 ^(a)) < 2k. Using the 
properties of 9 we know that ne i -(9^ m \a)) < 2 k, hence ne±(9( m )(a)) < 2k + 1. As nej.(0^ r "- ) (Q;)) > 2k, we 
must have nej.(0^ m ^(a)) = 2k + 1 and ne < _(0 ( ^ m ^(a)) = 2k. 

We have just showed that for a and 0W(a) belonging to A 2 k, 2 k+i, there exists m £ {1 ,..., i — 1} such 
that 0( m \a) £ A 2 k+i, 2 k- The proof is over if we manage to show the statement concerning a'. This can be 
done either by using the same reasoning or by an argument of cardinality (with Theorem 15). □ 


The previous lemma sets out how to build the desired involution. Essentially, from an involution of 
A 2 k+i, 2 k, we iterate 9 until obtaining an involution of A 2 k, 2 k+ 1 - If, on the other hand, the involution belongs 
to A 2 k, 2 k+i we want to go backward, so we iterate 9~ l until obtaining an involution of A 2 k+i, 2 k- If an 
involution does not fall under one of the previous forms, it necessarily belongs to a set of the form A(j, and 
we can then set this involution as a fixed point. 


Proposition 17. Let a be an involution and Aij be the set of involutions a such that nej.(a) = i and 
ne<_(a) =j. If a £ A 2 k, 2 k+ i, setm a as the smallest integer m such that 9^ (a) £ A 2 k+i, 2 k- If a. £ A2k+i,2k> 
set m a as the opposite of smallest integer m such that 9'-~ m \a) £ A 2 k, 2 k+ 1 - Otherwise, set m a as 0. 

The mapping a i— > 9^ ma \a) is an involution on the class of involutions that exchanges the statistics nei 
and ne<_. It preserves the size of the involutions, the number of fixed points and the opener/closer segwencau 

Remark. What about the open crossing level of an involution, that is to say the maximum number of dots 
contained in a ^-crossing, when this involution is transformed into an open matching? It is easy to see that 
the open crossing level shares the same distribution as the open nesting level or the enhanced nesting level 
(in particular via the bijection o r o 0). However, this statistic does not have a symmetric distribution, 
whether it is with the open nesting level or with the enhanced nesting level. 

4.3. Generating function expressions. One consequence of Theorem [T] is a collection of new generating 
function expressions for standard Young tableaux of bounded height. They come from an application of 
enumeration results of Weyl chamber walks mm- 


4.3.1. A Determinant Expression. The generating functions for Young tableaux of bounded height and Weyl 
chamber walks can both be expressed in terms of Bessel functions. We denote by 


b A x ) = / i( 2a; ) = 

n 


{2x) 2n+ i 
n\(n + j )\’ 


^if we consider isolated points as left endpoints of arcs 
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the hyperbolic Bessel function of the first kind of order j. 

Let Yk(t) be the exponential generating function for the class of standard Young tableaux with height 
bounded by k. Formulas for Y k {t) follow from works of Gordon, Houten, Bender and Knuth [T8HT91 I4]. which 
depend on the parity of k. We are only interested in the even values here. 

Theorem 18 ([TSj> fit, |j). The exponential generating function for the class of standard Young tableaux of 
height bounded by 2k is given by 

Y 2 k{t) det \bi—j(t) bi+j— \ (t)] i<i,j<.k • 

Around the same time, Grabiner-Magyar [20j determined a formula for the exponential generating function 
of the Wfc-oscillating walks of length n between two given points. Throughout this section, we translate the 
region W k to apply the previous results. Namely, we use 

W k = {(xi,..., x k ) e Z fc : x-x > ■ ■ ■ > x k > 0}. 

Theorem 19 (Grabiner-Magyar :2Tj]). For fixed A,/i £ Wk, the exponential generating function 0\^(t) of 
the W k -oscillating walks from A to /r, counted by their lengths, satisfies 

Ox^(t) = det (6 w _ A .(2f) - b lii+ x j (2t)) 1 ^. . 

We specialize the start and end positions as A = 5 = (k, k— 1,..., 1) and p. = S+me i = (k + m, k — 1,..., 1). 
We are interested in the sum over all values of to, and define O k {t) = J2 m >o Os,m ei +s(t)- We deduce the 
following. 

Proposition 20. The exponential generating function for the class of oscillating tableaux ending with a row 
shape is the finite sum 

k-1 2k—l—2u 

Ok{t) — 'y ' ( 1) y ~ bi dBt(bi—j bkd—i—j')o<.i<k—l,i^u,l< : j< : k—l- 

u=0 i—u 

This follows from the fact that the infinite sum which arises from direct application of Grabiner and 
Magyar’s formula telescopes after applying the identity b_ k = b k . The proof is technical, and largely an 
exercise in tracking indices after applying co-factor expansion of the determinants. 

The bijection between the classes implies O k (t) = Y 2 k(t). Here are the first two values, which are well 
known: 

Oi(t) = Y 2 (t) = b 0 + bi, 0 2 {t) = Y A {t) = bl + b 0 bi + b 0 b 3 - 2bib 2 - bj - b\ + bib 3 . 

4.3.2. A Diagonal Expression. Standard Young tableaux can also be viewed as oscillating tableaux with 
no deleting steps. (The entries tell you which box was added at a given time.) This gives us an interesting 
correspondence between two lattice path classes. 

Theorem 21. The set of oscillating lattice walks of length n in W k starting at 6 = (k, k — 1,..., 1) and 
ending at the boundary at the boundary {me\ + S : m > 0} is in bijection with the set of oscillating lattice 
walks of length n in W 2k , using only positive steps (ej), starting at S and ending anywhere in the region. 

We next obtain a new diagonal expression for standard Young tableaux of bounded height. The expres¬ 
sion is also a corollary of the bijection. We find the expression via the oscillating walks, and an application 
of Gessel and Zeilberger’s Weyl chamber reflectable walk model. The advantage of these diagonal repre¬ 
sentations is potential access to asymptotic enumeration formulas, and possibly alternative combinatorial 
representations. All of the generating functions are D-finite, and we can use the work of [5] to determine 
bounds on the shape of the annihilating differential equation. 

Theorem 22. The ordinary generating function for oscillating walks starting at 6 and ending on the bound¬ 
ary {mei + 8 : to > 0 }, is given by the formula 


O k {t ) = A 


t^-\z 3 zl--z^)(z 1 + l)Yl 




{Zi - Zj^ZiZj - 1) • n 2 <i<fcO? - !) 


1 - t{zx ■ ■ ■ z k ){zi + z 1 H - Zk + Z k ) 


The proof of Theorem [22] is a rather direct application of Gessel and Zeilberger’s formula for reflectable 
walks in Weyl chambers (the reader is directed to El7l(20j for details). 


13 





Proof. As oscillating tableaux are counted by walks in the region Wk := {(aq,..., Xd) : Xi > ■ ■ ■ > Xk > 0}, 
taking steps {±ei,... , ie*,}, and starting at 6 = (k, k — 1 ,... , 1 ), they are a reflectable walk model in the 
Weyl chamber Bk- This Weyl chamber is generated by the simple roots 


{ei — e 2 , ■ ■ ■, efc_i — efc, e*,}, 

and has a Weyl group G of order 2 k k\\ the group has the full action of ©j, on R fe , along with negating any 
subset of coordinates. Any group element er which acts on S by permuting its entries (keeping all its entries 
positive) has its order in the Weyl group the same as its order in &k- Furthermore, negating k entries of an 
element corresponds to an even element of the Weyl group if and only if k is even. Thus, if 

E(z) := {z\ ■ ■ ■ z k ) \\{zi - Zj), 

i<j 


then 

E :L)^)^) = £ (-i)' 7| £Mz)), 

wGG /C{l,...,fc} 

where oy sends the element Zj to 1 /zj if j £ I and fixes it otherwise. This sum simplifies to 

a(z)=£ ( —i)<m z ^) = * n & - -1) n (*? - !)> 

w£G ' ^ k -\<ro<rb 


l<j<i<k 


l<i<k 


which can be proven by noting that both expressions represent (zi • • • Zk) k A( z) as a polynomial of total degree 
k(3k + l)/2 with the same solutions (and then comparing the leading coefficients of both expressions). 

We are interested in walks that end on the boundary M = {me i + <5 : m £ N}. The generating function 
of the endpoints is 


B{ z) = 


E z_b 


i 


r fe-l fc-l fe-2 

/O-i /l -'0 /vo 


b elf *1 *2 ^3 —‘ z k(zi 1) 

Finally, the generating function of unrestricted walks starting at the origin is 


C{ z) 


1 

1 - t(zx ■ ■ ■ Zfc)(zi + Zi H- z k + Z k )' 


So by the classical result of Gessel and Zeilberger, we have the generating function representation 


F(t) := A (A(z)B(z)C(z)), 


which is exactly the formula given in the statement of this theorem. 


□ 


5. Tableau sequences as Baxter classes 


The combinatorial class that came to be known as Baxter permutations was introduced in 1967 in a 
paper of Baxter [2] studying compositions of commuting functions. A Baxter permutation of size n is 
a permutation a £ & n such that there are no indices i < j < k satisfying cr(j + 1) < cr(i) < u{k) < 
cr(j) or a(j) < a(k) < a(i) < cr(j + 1). We shall denote by B n the number of Baxter permutations of size 
n is B n . They constitute entry A001181 of the Online Encyclopedia of Integer Sequences (OEIS) [22] . 

Chung, Graham, Hoggart and Kleiman m found the explicit formula 


( 2 ) 


B„ 


n 


E 


(n+ 1\ f n + lN \ 

Vfc-1/ V k ) Vfc+1/ 

mm 


Many combinatorial classes have subsequently been discovered to have the same counting sequence - 
for example triples of lattice paths [2] and plane bipolar orientations [3j. A recent comprehensive survey 
of Felsner, Fusy, Noy and Orden |15| finds many structural commonalities among these seemingly diverse 
families of objects. Remarkably, there are intuitive bijections connecting these classes, see for instance [5]. 

The generating function of hesitating tableaux (i) in Proposition [7] was determined by Xin and Zhang |26| . 
Baxter numbers appear in their Table 3, and they mention that the equivalence between the two could be 
proved by applying creative telescoping to a formula for B n resembling the one given in Equation ([2]) above. 

Our contribution to this area is an explicit proof of that equivalence, and an exploration of the connection 
between these classes and the other well known Baxter classes. Clearly, the classes of Proposition [T] have 
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combinatorial bijections between them, but they do not share many of the properties of the other known 
Baxter classes. However, each of them does have a natural subclass of objects enumerated by Catalan 
numbers, as many Baxter families also do. (For example, non-crossing partitions are counted by Catalan 
numbers.) 

Proposition 23. The following classes are in bijection: 

(i) the set of hesitating tableaux with height bounded by 2, starting with empty diagram, ending in a 
partition with a single part; 

(ii) the set of open partition diagrams of length n with no enhanced 3- crossing; 

(in) \V 2 -hesitating walks of length n ending on the x-axis; 

(iv) Baxter permutations of size n + 1. 

Remark that Theorem [ 2 ] is simply the implication that (i) and (iv) from Proposition |23| are in bijection. 
We prove this with a generating function argument, and deduce the other bijections using Proposition [7] 

5.1. Proof of Theorem [2| We prove Theorem [2] and conjecture a stronger result which could be useful 
to prove the bijection combinatorially. This conjecture is partially verified using some of the intermediary 
computations, so it is useful to have them made explicit. We note that this is slightly different from both the 
proof that appears in a previous version of this work m and from the suggested proof of Xin and Zhang [26]. 

We first set up some notation. Let x = and consider the ring of formal series Q[x,x][t]. The opera¬ 
tor CT X extracts the constant term in x of series of Q[x, x][t]. 

We recall the work of Bousquet-Melou and Xin [7j. Here, we only require the k = 2 case from their work, 
and have consequently eliminated some of the subscripts from the statements of their results. Also, note 
that their definition of W 2 is shifted one unit to the right, hence in the statement of their results, walks start 
at (1,0) rather than (0,0). 

Let Q denote the first quadrant in the plane, Q = {(x,y) : x,y > 0}, and let W 2 denote the region 
W 2 = {(x,y) : x > y > 0}. Walks taking n steps that start at A and end at p and remain in Q and W 2 are, 
respectively, denoted by q(X,p,n) and w(X,p,n). 

Bousquet-Melou and Xin’s Proposition 12 in [7], based on a classic reflection argument, implies the 
following. For any starting and ending points A and p in W 2 , the number of RVhesitating walks going from 
A to p can be expressed in terms of the number of Q-hesitating walks: 

w( A, p, n) = q( A, p, n) - q{ A, p, n ) 

where ( x,y ) = ( y,x ). They define a simple sign reversing involution between pairs of walks; the walks 
restricted to W 2 appear as fixed points. 

We consider the following two generating functions for Q-hesitating walks that start at (1,0) and end on 
an axis: 

H(x;t) = Y ^((i) 0)i (b 0), 2n)x l t n and V(y;t) = Y °). (M> 2n)y l t n . 

i>l,n>0 i>l,n>0 

By applying the proposition we see immediately that the bivariate generating function W (x; t) for Un¬ 
hesitating walks that start at (0,1) and end on the x- axis satisfies the formula 

(3) W{x\t)= Y^ ^((1,0), (i, 0), 2n) x z t n = H(x; t) — V(x\ t). 

i>l,n>0 

Theorem [2] is equivalent to the statement 

( 4 ) W(l;t) = Y B n+it n - 

Proof of Theorem [1| For i > 0, Bousquet-Melou and Xin [7] show the following 

(5) [x i+1 \H(x-t) = CT,— 1 —-x 2+i (x 2 -x 2 T 2 + x 3 F) 

t y 1 — I - X J 

(6) [x l+1 ]V(x-,t) = CT Y x 3+i {x 2 -x 2 Y 2 +x 3 Y). 

t(l + x) 
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Thus, we deduce 


\x i+1 ]W{x-,t) = [x i+1 ]H(x;t ) - V(x-,t) = CT 
Hence we have 


Y 


2+i ^3+j\/„,2 ^.2\r2 _j_ ^3 


t{ 1 + X) ' 

W(l-,t) = T(t) + U(t) + V(t), 


x —x nx — x 


Y 2 + x A Y). 


where 




y3 


tf(i) = -CT 


i>0 


t(l + X) 


H(t) = CT,y 


y2 


i>0 


t( 1 + X) 


(x 5+I 


Then we use the following identity from [7] (valid for k > 1 and £ £ Z) 


CT, 


y A 


t(l + a;) 

which gives the following simplifications: 


r = ct. 


yA 


/-A+l 


£(1 + x)‘ 


T(t ) = CT,^ ^ | ^ (^ i -a: 5+, )=CT I ^ l | ^ (1 + z + a : 2 + z 3 + j 4 ), 


i>0 


*( 1 - 


' f(l + x) 


U(t) = -CT, y - x 1+i ) = CT y 


i>0 


t(l + x) 


7 X , 


V(t) = CT, y ^ (x 4+i - X*) = CT. 12 


i>0 


t(l + x) 


t( 1 + x) 


t( 1 + a;)‘ 

(—1 — x — x 2 — x 3 ). 


Hence, defining Af f-^t) = CT ^^ ^, we can collect terms to obtain 

4 3 

(7) bb(l; t) = y A-,i(i) + ^ 1,3 (i) ~ y A-,2(i)- 


r—0 


r—0 


It is shown in .7] that the Lagrange inversion formula yields, for n £ N, 

k fn+\\fn + l 


[t n ]Ae t k(t) = y a n {£, k,j), with a n (£, k,j) = 


n + 1 V 3 ) \3 + k J \3 - 


Here we apply the convention (™) = 0 for j < 0 or j > n. 

Next, it is straightforward to detect and check the linear relations (valid for n £ N and j £ Z) 

a n { 4, l,n — j + 2) + a n (l,3,j - 1) - a n (2,2,n- j + 1) - a„(3,2, j) = 0, 

a n (l, 1 ,n-j) + a„(2,1, j + 1) - a n (0,2, j) = 0, 

which respectively give Ai^{t)-\-A 3 ^{t) — A 2 , 2 (£)—A 3i 2 (£) = 0 and Ai t i(t)+A 2 .i(t)—A 0 ^{t) = 0. Remarkably, 
expression |7]) for W{l\t) simplifies to 

W(t) = Ho,i(£) + A 3yl (t) — Hi j2 (£)- 

("+ 1 )("+ 1 )(“+ 1 ) 

For n > 1, the Baxter number B n is given by B n = Lijez^nji with b n j = ^V+iwn+ij 1 ' 1 '' , and again it is 

easy to detect and check that (for n £ N and j £ Z) 

1 1 j) T a n (3,1, j T 1) u n (1,2, j) = &n+i,j+i, 

so that H 0) i(£) + -4 3 ,i(£) - A ia {t) = J 2 n >o B n+1 t n , and thus [t n ]W(l\t) = □ 
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{[0,0]; [i, j] -> 


[*,»], [* + !>J] 

[*. j]> [*,J + !]> - ••>[*, * - 1], ifi > 0 

[i - 1, j], [i - l,j + 1],.,. , [i - l,i - 1], ifi > 0 

[i, j — 1] , [i — 1, j — 1] if i > 0, and j > 0}. 


Figure 13. The first five levels of each of the Baxter generating trees. They are respectively 
from 0 0 Mr 


5.2. Consequence: a new generating tree. A generating tree for a combinatorial class expresses recursive 
structure in a rooted plane tree with labeled nodes. The objects of size n are each uniquely generated, and 
the set of objects of size n comprise the nth level of the tree. They are useful for enumeration, and for 
showing that two classes are in bijection. Theorem [2] yields a new generating tree construction for Baxter 
objects. 

Several different formalisms exist for generating trees, notably pQ. The central properties are as follows. 
Every object 7 in a combinatorial class C is assigned a label £( 7 ) £ Z fe , for some fixed k. There is a 
rewriting rule on these labels with the property that if two nodes have the same label then the ordered list 
of labels of their children is also the same. We consider labels that are pairs of positive integers, specified 
by {^Root : [i,j] —> Succ([i, j])}, where ^R 00 t is the label of the root. 

Two generating trees for Baxter objects are known in the literature, and one consequence of Theorem [2] 
is a third, using the generating tree for open partitions given by Burrill et al. |10| . This tree differs from the 
other two already at the third level, illustrating a very different decomposition of the objects. For the three 


different systems we give the succession rules, and the first 5 levels of the tree (unlabelled), in Figure 13 


5.3. A conjectured refinement. We have proved that the coefficients a(n,m) counting BVhesitating 
walks of length 2 n from (0,0) to (to, 0) satisfy J2m a ( n ’ m ) = Bn+i, with B n the nth Baxter number. A 
bijective proof is yet to be found, and in that perspective a natural question is whether the parameter to 
corresponds to a simple parameter on another Baxter family. 

Proposition 24. The family of Q-hesitating excursions, that is to say the hesitating walks in the lattice 
Q = {{x,y) : x,y > 0} starting and ending at the origin, is a Baxter family: the number of such walks of 
length 2 n is equal to B n+ 

Proof. We show an easy bijection with the set T n of non-intersecting triples of lattice paths each of length n 
with steps either N = (0,1) (north steps) or E = (1,0) (east steps), with respective starting points (—1,1), 
(0, 0), (1, —1) and respective ending points (fc—1, n—k+ 1), (k, n—k ), (fc+1, n—k—l) for some k £ {0,..., n}. 
For 3 distinct points Pi,P2,P3 in on a same line of slope —1, ordered from top-left to bottom-right, define 
the distance-pair for {pi,P 2 ,Ps) as the pair (i,j) of nonnegative integers such that x(pi) = x(j> 2 ) — i — 1 and 
x(p 3 ) = x{p 2 ) + j + 1. Let (Pi, P 2 , P3) £ T n - For r £ {0,..., n} and z £ {1,2, 3}, let p^ be the point on P z 
after r steps, and let d[r) be the distance-pair for (p^\p^\p^)', note that d(0) = (0,0) and d{n) = (0,0) 
and that d(r) £ q for 0 < r < n. Moreover, for 0 < r < n, the vector <5(r) := d(r) — d(r — 1) is in the 
set {(±1,0), (0, ±1), (1, —1), (—1,1), (0, 0)}, with two possibilities for being (0, 0) (whether the rth steps in 
Pl, P2, P3 are all north or all east). Hence the situation for the successive distance-pairs do,... ,d n is exactly 
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the same as for the successive points of even rank in a Q-excursion of length 2 n. Figure [l4] (left and middle) 
illustrates this bijection. □ 





Figure 14. Left. A non-intersecting triple of lattice paths. Middle. A Q-hesitating excur¬ 
sion. The stay steps are drawn as loops. The switch-multiplicity of the walk is 3 (the white 
arrows indicate the marked steps). Right. A W^-hesitating excursion with 3 marked steps 
each leaving the diagonal. These three objects are in correspondence. 


We now define a secondary parameter to for Q-excursions. Let w be a Q-excursion of length 2 n where e r 
denotes the rth step, for 1 < r < 2 n. Consider, if any, the first step that visits the region x < y. Then 
consider, if any, the first step ej 2 after eq that visits the region x > y, and so on (switching between x < y 
and x > y each time). We have here a stopping iterative process yielding, for some m > 0, m marked steps 
ei lt ..., ei m with i\ < ... < i m ; m is called the switch-multiplicity of the excursion. For instance, the switch- 
multiplicity of the excursion at the middle of Figure [T4| is 3. Note also that m < n since two marked steps 
cannot be consecutive (the case m = n is reached by the unique excursion where i\ = 1, *2 = 3, = 5,..., 

i.e., the excursion that alternates pairs of steps (0,1), (0,-1) with pairs of steps (1, 0), (—1, 0)). Denote 
by g(n,m) the number of Q-hesitating excursions of length 2 n and switch-multiplicity m, and a(n, m) the 
number of WVhesitating walks of length 2 n from (0, 0) to (m, 0). 

Conjecture 1. For n^m > 0, we have q(n,m ) = a(n,m). 

We have thought of the switch-multiplicity as a natural candidate because of the analogy with a well- 
known bijection between excursions of length 2 n on the line Z and walks of length 2 n starting at 0 on the 
half-line Z>o (with steps in ±1 for both types of walks), where a similar switch-multiplicity parameter for 
excursions (this time switching between Z<o and Z>o) corresponds to half the ending abscissa of positive 
walks. However, what surprises us is that, while we have strong evidence the conjecture is true, we do not 
even have a proof for m = 1 (the case m = 0 is trivial). 

Let us now slightly reformulate the conjecture so that we have HVhesitating walks on both sides. Consider 
a Q-hesitating excursion w of length 2n, with eq,..., Ci m the steps given by the stopping iterative process 
(switching between x < y and x > y). Accordingly w splits into a concatenated sequence of m + 1 parts 
7 Tq,... ,7r m , where 7To is the part before eq (7To = w if m = 0), for 1 < h < to, 7r^ is the part between ei h 
(included) and ei h+1 (excluded), and for m > 1, 7r m is the ending part of H'j starting from e,; m . Each walk 
7Ti starts and ends on the diagonal x = y and stays in x > y for i even and in x < y for i odd. Hence, if we 
reflect each odd walk 7T2i+i according to the diagonal x = y, we obtain a WVhesitating walk from (0, 0) to 
(0, 0) with to marked steps (the steps at positions i i,..., i m ) each leaving the diagonal x = y. In addition, 
due to recording the marked steps, there is no loss of information (the original excursion can be recovered). 
This correspondence is depicted by Figure [l4| (middle and right). Hence, if we denote by a(n;i,j,m ) the 
number of W^-hesitating walks of length 2 n from (0,0) to ii,j) with 2 n steps and to marked steps each 
leaving the diagonal x = y, Conjecture [l] can be reformulated as: 

Conjecture 2 (Reformulation). Forn,m > 0, we have 

a(n ; m, 0, 0) = a(n ; 0,0, to). 
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Actually, an even stronger symmetry seems to hold: 

Conjecture 25. For n,i,j > 0, we have 

a(n-,i,0,j) = a{n\ j, 0, i). 

Note that there is clearly a one-to-one correspondence between steps leaving the diagonal x = y and 
steps ending at the diagonal x = y. Hence a(n; i,j, to) is also the number of BVhesitating walks of length 
2 n from (0,0) to (i,j) with 2 n steps and m marked steps each ending at the diagonal x = y. In that 
form it is easy to obtain a recurrence for the coefficients a(n;i, j, m) by considering the effect of adding the 
last two steps (note that each of the two last steps has to be unmarked if empty or not ending at x = y, 
and might be either unmarked or marked if non-empty and ending at x = 1). Denote by S the set of steps 
{(±1, 0), (0, ±1), (—1,1), (1, —1)} together with the two stationary steps Si and s 2 , where Si simulates taking 
the pair (1,0) and (—1,0) as a single step, and s 2 simulates taking the pair (0,1) and (0,-1) as a single 
step. We have the following recurrence (with 5 the Kronecker symbol), from which we have been able to 
check that Conjecture [25] holds for all n,i,j with n < 56: 

• for n = 0, 

a(n;i,j,m) = 1 if i = j = m = 0, 

a(n;i,j,m) = 0 otherwise, 

• for n > 0, 


0 for (■i , j , to) 

tv, 

*«-» 

VI 

o 

II 

< i, 0 < to < n} 

S i=j ■ °( n 

- M 

- x(s),j 

- y(s), m) 





+Si>j ■ y, a ( n ■ 

-1; i- 

- x(s),j - 

- y(s),m) 

s£<S 




+ § i=j ■ Y a ( 

n — 1; 

i — x(s), 

j — y(s), to — 1) 


sG«S\si 

+6i-j +1 • a(n — 1 1) for (■ i , j, to) £ V. 


6. Conclusion 

We conclude with a few thoughts on future directions. Starting with the correspondences described here, 
we can easily give an expression for the generating function of standard Young tableaux of bounded height 
as a diagonal of a rational function. The original proofs of the D-finiteness of these generating functions 
(for arbitrary k) of Gessel [T5| used a diagonal type operation over series with infinite variable sets. We 
are interested in the conjecture of Christol which states that globally bounded D-finite series can always be 
expressed as a diagonal of a rational function. Perhaps the key to understanding how to find such diagonal 
expressions lurks in Weyl chamber walk representations of combinatorial objects, since they easily yield 
diagonal expressions from the machinery of Gessel and Zeilberger. 

Baxter numbers generalize, in some sense, Catalan numbers. Both are ubiquitous combinatorial sequences, 
and both are related to hesitating walk families. Are hesitating walks in higher dimensions similarly common? 

Acknowledgements 

We are extremely grateful to Sylvie Corteel, Lily Yen, Yvan le Borgne, Sergi Elizalde, and Guillaume 
Chapuy for stimulating conversations, and important insights. JC is supported by the ANR GRAAl , ANR- 
14-CE25-0014-02, and by the PIMS postdoctoral fellowship grant. The work of EF was partly supported by 
the ANR grant Cartaplus 12-JS02-001-01 and the ANR grant EGOS 12-JS02-002-01. SM is supported by 
an NSERC Alexander Graham Bell Canada Graduate Scholarship. The work of MM is partially supported 
by an NSERC Discovery Grant. 


19 


References 


[1] Cyril Banderier, Mireille Bousquet-Melou, Alain Denise, Philippe Flajolet, Daniele Gardy, and Dominique Gouyou- 
Beauchamps. Generating functions for generating trees. Discrete Math., 246(l-3):29-55, 2002. Formal power series and 
algebraic combinatorics (Barcelona, 1999). 

[2] Glen Baxter. On fixed points of the composite of commuting functions. Proc. Amer. Math. Soc., 15:851-855, 1964. 

[3] R. J. Baxter. Dichromatic polynomials and Potts models summed over rooted maps. Ann. Comb., 5(l):17-36, 2001. 

[4] Edward A. Bender and Donald E. Knuth. Enumeration of plane partitions. J. Combinatorial Theory Ser. A, 13:40-54, 
1972. 

[5] Nicolas Bonichon, Mireille Bousquet-Melou, and Eric Fusy. Baxter permutations and plane bipolar orientations. Sem. 
Lothar. Combin., 61A:Art. B61Ah, 29, 2009/11. 

[6] Alin Bostan, Pierre Lairez, and Bruno Salvy. Creative telescoping for rational functions using the Griffiths-Dwork method. 
In Proceedings of the International Symposium on Symbolic and Algebraic Computation (ISSAC), New York, NY, USA. 
ACM., pages 93-100, 2013. 

[7] Mireille Bousquet-Melou and Guoce Xin. On partitions avoiding 3-crossings. Sem. Lothar. Combin., 54:Art. B54e, 21 pp. 
(electronic), 2005/07. 

[8] Mathilde Bouvel and Olivier Guibert. Refined Enumeration of Permutations Sorted with Two Stacks and a Ds-Symmetry. 
Ann. Comb., 18(2):199-232, 2014. 

[9] Sophie Burrill. A generating tree approach to k-nonnesting arc diagrams. PhD thesis, Simon Fraser University, Burnaby, 
Canada, July 2014. http://summit.sfu.ca/system/files/iritemsl/14390/etd8479_SBurrill.pdf. 

[10] Sophie Burrill, Sergi Elizalde, Marni Mishna, and Lily Yen. A generating tree approach to k-nonnesting partitions and 
permutations. arXiv preprint arXiv:1108.5615, 2012. 

[11] Sophie Burrill, Melczer, Stephen, and Marni Mishna. A Baxter class of a different kind. In Proceedings of the 27th Inter¬ 
national Conference on Formal Power Series and Algebraic Combinatorics, Korea, 2015. 

[12] William Y. C. Chen, Eva Y. P. Deng, Rosena R. X. Du, Richard P. Stanley, and Catherine H. Yan. Crossings and nestings 
of matchings and partitions. Trans. Amer. Math. Soc., 359(4):1555-1575 (electronic), 2007. 

[13] F. R. K. Chung, R. L. Graham, V. E. Hoggatt, Jr., and M. Kleiman. The number of Baxter permutations. J. Combin. 
Theory Ser. A, 24(3):382-394, 1978. 

[14] Serge Dulucq and Olivier Guibert. Baxter permutations. In Proceedings of the 7th Conference on Formal Power Series 
and Algebraic Combinatorics (Noisy-le-Grand, 1995), volume 180, pages 143-156, 1998. 

[15] Stefan Felsner, Eric Fusy, Marc Noy, and David Orden. Bijections for Baxter families and related objects. J. Combin. 
Theory Ser. A, 118(3):993-1020, 2011. 

[16] Ira M. Gessel. Symmetric functions and P-recursiveness. J. Combin. Theory Ser. A, 53(2):257-285, 1990. 

[17] Ira M. Gessel and Doron Zeilberger. Random walk in a Weyl chamber. Proc. Amer. Math. Soc., 115(1):27—31, 1992. 

[18] Basil Gordon. Notes on plane partitions. V. J. Combinatorial Theory Ser. B, 11:157-168, 1971. 

[19] Basil Gordon and Lome Houten. Notes on plane partitions. I, II. J. Combinatorial Theory, 4:72-80; 81-99, 1968. 

[20] David J. Grabiner and Peter Magyar. Random walks in Weyl chambers and the decomposition of tensor powers. J. Algebraic 
Combin., 2(3):239-260, 1993. 

[21] Olivier Guibert and Svante Linusson. Doubly alternating Baxter permutations are Catalan. Discrete Math., 217(1-3):157- 
166, 2000. Formal power series and algebraic combinatorics (Vienna, 1997). 

[22] OEIS Foundation Inc. The On-Line Encyclopedia of Integer Sequences. Published electronically at http://oeis.org, 2011. 

[23] Christian Krattenthaler. Growth diagrams, and increasing and decreasing chains in fillings of Ferrers shapes. Adv. in Appl. 
Math., 37(3):404-431, 2006. 

[24] Christian Krattenthaler. Bijections between oscillating tableaux and (semi)standard tableaux via growth diagrams. 
arXiv: 1412.5646 [math.CO], 2014. 

[25] M.-P. Schiitzenberger. La correspondance de Robinson. In Combinatoire et representation du groupe symetrique (Actes 
Table Ronde CNRS, Univ. Louis-Pasteur Strasbourg, Strasbourg, 1976), pages 59-113. Lecture Notes in Math., Vol. 579. 
Springer, Berlin, 1977. 

[26] Guoce Xin and Terence Y. J. Zhang. Enumeration of bilaterally symmetric 3-noncrossing partitions. Discrete Math., 
309(8):2497-2509, 2009. 

S. Burrill, Department of Mathematics, Simon Fraser University, Burnaby BC Canada. 

J. Courtiel, Department of Mathematics, Simon Fraser University, Burnaby BC Canada. 

E. Fusy, LIX, Ecole Polytechnique, Palaiseau, France. 

S. Melczer, Cheriton School of Computer Science, University of Waterloo, Waterloo ON Canada. & U. Lyon, 
CNRS, ENS de Lyon, Inria, UCBL, Laboratoire LIP 

M. Mishna, Department of Mathematics, Simon Fraser University, Burnaby BC Canada. 


20 



